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5.3.1 Covariance and Correlation 

Consider two random variables X and Y. Here, we define the covariance between X and Y , written Cov( X, Y ) . The covariance gives 
some information about how X and Y are statistically related. Let us provide the definition, then discuss the properties and applications of 
covariance. 

The covariance between X and Y is defined as 

Cov(X,E) = E[(X - EX)(Y - EY)\ = E[XY] - (EX)(EY). 

Note that 

E[(X - EX)(Y - EY)\ = E[XY — X{EY) - {EX)Y + {EX){EY)\ 

= E[XY] - ( EX)(EY) - (EX)(EY) + (EX)(EY) 

= E[XY] - (EX)(EY). 

Intuitively, the covariance between X and Y indicates how the values of X and Y move relative to each other. If large values of X tend to 
happen with large values of Y , then (V — EX) (Y — EY) is positive on average. In this case, the covariance is positive and we say X 
and Y are positively correlated. On the other hand, if X tends to be small when Y is large, then ( X — EX) (Y — EY) is negative on 
average. In this case, the covariance is negative and we say X and Y are negatively correlated. 

Example 5.32 

Supposed ~ Uniform(l, 2), and given X = X,Y is exponential with parameter A = X. Find Cov(V, V). 

• Solution 

o We can use Cov( X, Y) — EXY — EXEY . We have EX — y and 

EY = SfE’fyjX]] (law of iterated expectations (Equation 5.17)) 

(since Y\X ~ Exponential(X )) 



= In 2. 

We also have 

EXY — E[E[XY\X]] (law of iterated expectations) 

EXY = E[XE[Y\X]\ (smceE[X\X = x] = x) 

(since Y\X ~ Exponential(X )) 

= 1 . 


= E 


1 



Thus, 


Cov(X,y) = E[XY] - (EX)(EY) =1- |ln2. 


Now we discuss the properties of covariance. 

Lemma 5.3 

The covariance has the following properties: 

1. Cov(X,X) = Var(X); 

2. if X and Y are independent then Cov(J^, E) = 0; 

3 . Cov(X,E) = Cov(E,X) : 

4 . Cov(aX, Y) = aCov(X, Y)- 

5 . Cov(X +c,Y)= Cov(X, Y); 

6. Cov(X + Y, Z) = Cov(X, Z) + Cov(E, Z)- 
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7. more generally, 

( m n \ m n 

Y,».x„Y, b iYj =£E a t b ] Gov(X l ,Yj). 

i =1 i=i / 1=1 i=i 

All of the above results can be proven directly from the definition of covariance. For example, if X and Y are independent, then as we have 
seen before E [.XV] = EXEY, so 

Cov(X, Y) = E[XY] - EXEY = 0. 

Note that the converse is not necessarily true. That is, if Cov( X, Y ) = 0, X and Y may or may not be independent. Let us prove Item 6 
in Lemma 5.3, Cov(X + Y, Z) — Cov(X, Z ) + Cov(V, Z). We have 

Co v(X + Y,Z) = E[(X + Y)Z] — E(X + Y)EZ 
= E[XZ + YZ\ - (EX + EY)EZ 
= EXZ - EXEZ + EYZ - EYEZ 
= Cov(X, Z) + Cov(Y, Z ). 

You can prove the rest of the items in Lemma 5.3 similarly. 

Example 5.33 

Let X and Y be two independent N (0, 1) random variables and 

Z = 1 + X+XY 2 , 

W = 1 + X. 

Find Cov(^, W). 

• Solution 

Co v(Z,W) = Cov(1+X + XY 2 ,1 + X ) 

= Co v(X+XY 2 ,X) 

= Cov(X, X) + Cov(Xy 2 , X) 

= Var (X) + E[X 2 Y 2 } - E[XY 2 ]EX 
= 1 + E[X 2 ]E[Y 2 } - E[X] 2 E[Y 2 } 

= 1 + 1 - 0 = 2 . 


(by part 5 of Lemma 5.3) 

(by part 6 of Lemma 5.3) 

(by part 1 of Lemma 5.3 & definition of Cov) 
(since X and Y are independent) 


Variance of a sum: 

One of the applications of covariance is finding the variance of a sum of several random variables. In particular, if Z — X + Y, then 

Var (Z) = Co v(Z, Z) 

= Cov(x + y,x + y) 

= Cov(X, X) + Cov(X, Y) + Cov(y, X) + Cov(y, Y) 

= Var(X) + Var(y) + 2Cov(X, Y). 

More generally, for a, b £ M, we conclude: 

Var(aX + bY) = a 2 Var(X) + 5 2 Var(y) + 2abCov(X, Y) (5.21) 


Correlation Coefficient: 

The correlation coefficient, denoted by pxY or p(X, Y) , is obtained by normalizing the covariance. In particular, we define the correlation 
coefficient of two random variables X and Y as the covariance of the standardized versions of X and Y. Define the standardized versions 
of X and Y as 
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X-EX Y-EY 

U= -, V= - 


ox 


oy 


Then, 


PxY = Cov(U, V ) = Cov 


X-EX Y-EY 


ax ay 

— Cov ( 

,Ox cry J 


_ Cov(X,y) 

axay 

Pxy — p{X, Y) 


(5.22) 


(by Item 5 of Lemma 5.3) 


Cov(x,y) _ Cov(x,y) 


y / Var(X) Var(Y) 


axay 


A nice thing about the correlation coefficient is that it is always between — 1 and 1. This is an immediate result of Cauchy-Schwarz 
inequality that is discussed in Section 6.2.4. One way to prove that — 1 < p < 1 is to use the following inequality: 

a 2 + B 2 

a/3 < -, for a, /3 E IK. 


This is because (a — /3) 2 > 0- The equality holds only if a = /3. From this, we can conclude that for any two random variables U and 

V. 


E[UV] < 


EU 2 +EV 2 


with equality only if U = V with probability one. Now, let U and V be the standardized versions of X and Y as defined in Equation 
5.22. Then, by definition pxy — Cov(U, V) = EUV. But since EU 2 = EV 2 — 1, we conclude 

r , eu 2 + ev 2 

PX y = E[UV] < ---= 1, 


with equality only if U = V . That is. 


Y-EY X-EX 


ay 


Ox 


which implies 

(eY - —EX 
\ ox 

where a and b are constants. 

p(-X,Y) < 1. 

But p(— X, y) = -p(X, Y), thus we conclude p(X, Y) > — 1. Thus, we can summarize some properties of the correlation 
coefficient as follows. 

Properties of the correlation coefficient: 

1. -1 < p(X,Y) < 1; 

2. if p(X, Y) — 1, then Y — aX + b, where a > 0; 

3. if p(X, Y) — — 1, then Y = aX + b, where a < 0; 

4. p(aX + b, cY + d) — p(X, Y) for a, c > 0. 

Definition 5.2 

Consider two random variables X and Y : 

- If p(X, Y) — 0, we say that X and Y are uncorrelated. 


CTv 

Y = —X + 
ox 

= aX + b, 


Replacing X by —X, we conclude that 
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-if P(X,Y) 
-if P(X,Y) 


> 0, we say that X and Y are positively correlated. 
< 0 , we say that X and Y are negatively correlated. 


Note that as we discussed previously, two independent random variables are always uncorrelated, but the converse is not necessarily true. 
That is, if X and Y are uncorrelated, then X and Y may or may not be independent. Also, note that if X and Y are uncorrelated from 
Equation 5.21, we conclude that Var(^C + Y) = Var(JC) + Var(y). 

If X and Y are uncorrelated, then 


Var(X + Y) = Var(X) + Var(Y). 


More generally, if X\ , X^,. . . , X n are pairwise uncorrelated, i.e., p{X^ , Xj ) = 0 when i ± j, then 


Var(Xi + X 2 +... +X n ) = Var(Xi) + Var(X 2 )+... +Var(X n ). 


Note that if X and Y are independent, then they are uncorrelated, and so Var(Ji + Y) = Var(^) + Var(Y). This is a fact that we 
stated previously in Chapter 3. and now we could easily prove using covariance. 


Example 5.34 

Let X and Y be as in Example 5.24 in Section 5.2.3, i.e., suppose that we choose a point (^l, Y) uniformly at random in the unit disc 

D = {(x,y)\x 2 +y 2 < 1 }. 

Are X and Y uncorrelated? 


• Solution 

o We need to check whether Cov(^l , Y) = 0. First note that, in Example 5.24 of Section 5.2.3 . we found out that X and Y 
are not independent and in fact, we found that 

X\Y - Uniform(-y/l-Y 2 ,y/l-Y 2 ). 

Now let's find Cov(JC, Y) = EXY — EXEY . We have 

EX — U|U[X| Y]] (law of iterated expectations (Equation 5.17)) 

= E[0] = 0 (since X\Y - Unifarm(-y/l - Y 2 , y/\ - Y 2 )). 

Also, we have 

E[XY] = E[E[XY\Y]] (law of iterated expectations (Equation 5.17)) 

= E[YE[X\Y}\ (Equation 5.6) 

= E[Y ■ 0] = 0. 

Thus, 


Cov(X, Y) = E[XY] - EXEY = 0. 

Thus, X and Y are uncorrelated. 


<— previous 
next —> 
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